Learning Points & Summary¶
In this lesson, we reviewed many concepts that we've covered previously, including:
Create a rolling average to smooth out time-series data and show a trend, e.g. prizes_per_year.rolling(window=5).mean()
How to use .value_counts(), .groupby(), .merge(), .sort_values() and .agg(). Note: sometimes .value_counts() more elegant than .groupby().agg({'column': 'count'})
Summary of all the import statements:
import pandas as pd
import numpy as np
import plotly.express as px
import seaborn as sns
import matplotlib.pyplot as plt
Using the .transpose() or the .T to get the transposed view of a DataFrame
Using .insert(): df_data.insert(loc=5, column='share_pct', value=numerator/denominator)
Using .drop([labels_list], axis='columns') when dropping columns (and axis="rows" by default otherwise)
Using .duplicated() with parameters, e.g. df_data.duplicated(subset=['full_name'], keep=False)
Using .nunique() to find number of unique values in a Series
Using .reset_index, for example after dropping values: df_cumul_country.dropna().reset_index(drop=True)
Using .dt to access datetime attributes or methods, applied to a Series, e.g. df.birth_year.dt.year
Using .describe() to return "Descriptive Statistics" about a DataFrame
Using astype() to convert a Series to a datatype, e.g. .astype('int64')
Creating plotly donut charts with .pie() and some parameters like hole=0.5, height=500, title=
Creating plotly bar charts with .bar() and parameters like labels={'old': 'new'}, color_continuous_scale='Aggrnyl', color=category, as well as orientation='h' to set the bars horizontal (consider the new x and y params in this case)
Customizing plotly charts with .update_traces() and parameters like textfont_size=15, coloraxis_showscale=False
Changing the order of the bars in plotly with fig3.update_xaxes(categoryorder='total descending')
Creating regression plots with seaborn and .regplot() and changing the style of the scatter plot with scatter_kws={} and of the reg line with line_kws={}, e.g.
scatter_kws={'s':8, 'alpha': 0.4, 'color': '#2f4b7c'}, # s for the size of the dots
line_kws={'color': 'brown', 'linewidth': 2}
- Using matplotlib mainly for graphs with several y-axes with
.plot() and .scatter, and using all the parameters and methods such as:
plt.figure(figsize=(16,8), dpi=200) # setting the size and resolution for display
plt.xticks(ticks=x_ticks, fontsize=14, rotation=45) # styling the x-ticks
plt.yticks(fontsize=14) # styling the y-ticks
plt.title("Number of Nobel Prizes per Year", fontsize=18) # give a title
plt.xlabel('Year', fontsize=16) # styling the x-label
plt.ylabel('Number of Prizes', fontsize=16, color='crimson') # styling the x-label
plt.xlim(1900, 2020) # setting the x-axis boundaries
ax1 = plt.gca() # Get Current Axes
ax2 = plt.twinx() # Creating a second y-axis (which shares the same x-axis)
ax1.scatter(prizes_per_year.index, prizes_per_year.prize, c='dodgerblue',
alpha=0.6, s=100) # c for color, alpha transparency, s marker size
ax1.plot(prizes_per_year.index, prizes_per_year_rolling, c='crimson',
linewidth=3) # increasing linewidth
ax1.grid(color='grey', linestyle='--') # adding a grid
# Third positional argument in plot is fmt = '[marker][line][color]'
ax2.plot(prizes_per_year.index, prize_share_rolling.share_pct, '--g', # dashed line green
linewidth=3)
ax2.set_ylim(0, 1)
ax2.set_ylabel('Prize Share', fontsize=16, color='green')
plt.setp(ax3.get_yticklabels(), fontsize=14)
ax2.invert_yaxis()
plt.show()
In addition, we learned many new things too. We looked at how to:
- Using shell commands with the
! or % characters (see saved links in folder "IPython magic commands ! $ % etc - run shell commands"), e.g.
!pip install --upgrade plotly
- Create a subset separately and then re-use it, e.g.
col_subset = ['year','category', 'laureate_type',
'birth_date','full_name', 'organization_name']
df_data.loc[df_data.birth_date.isna()][col_subset] # we can reuse a subset like this!
- The
.str.split() method returns a Series where values are lists of two items such as [1, 1] (unless we use the expand=True parameter). Then we use .map() or .apply() and give it a lambda function to get hold of the first or second element. See StackOverflow post. E.g.
numerator = df_data.prize_share.str.split('/').map(lambda x: x[0]).astype('int64')
denominator = df_data.prize_share.str.split('/').map(lambda x: x[1]).astype('int64')
- Use negation in the
.loc[] conditions (in general with pandas and numpy, we use the bitwise NOT ~ instead of ! or not):
df_people = df_data.loc[not df_data.sex.isna()] # where isna not True -> NOK
df_people = df_data.loc[~ df_data.sex.isna()] # OK
- Select a specific part of a DataFrame using
.iloc[] -> df.iloc[row_start:row_end , col_start, col_end], e.g.
Use .fillna() to replace NaN values with a specific value, e.g. category_and_gender.fillna(value='Organization', inplace=True)
Using Numpy .arange() method to create a custom array of ticks for our graph. Also knowing that np.random.randint(0, 5, 10) returns 10 random values between 0 included and 5 excluded.
x_ticks = np.arange(1900, 2021, 5)
plt.xticks(ticks=x_ticks, fontsize=14, rotation=45)
- Use
.isin() to check values of a Series that are in another Series, e.g.
df_data.loc[df_data.birth_country_current.isin(top20_countries.birth_country_current)]
- Use the
.cumcount() method wisely. It only returns a Series, and the count starts at 0, e.g.
cumul_prize_country_series = df_data_sorted.groupby('birth_country_current').cumcount()
df_cumul_country['Cumulated Number of Nobel Prizes'] = df_cumul_country['Cumulated Number of Nobel Prizes'] + 1
- Use the
.copy() method when/if getting a warning about trying to create a column, e.g.
df_cumul_country = df_data_sorted[['year', 'birth_country_current']].copy()
- Use
.count() and then .cumsum(), and the ability to group by an index level, as an alternative to .cumcount(), e.g.
prize_by_year = df_data.groupby(by=['birth_country_current', 'year']).agg({'prize': 'count'})
cumulative_prizes = prize_by_year.groupby(level=[0]).cumsum()
Use float formatting in an f-string, e.g. print(f"blabla the age is {df_data.winning_age.mean():.0f} years old.")
Force-display several elements in the same block of code, without using print with : display()
- Select the N largest or N smallest of a DataFrame with
.nlargest(), .nsmallest()
display(df_data.nlargest(n=1, columns='winning_age'))
display(df_data.nsmallest(n=1, columns='winning_age'))
Create a Choropleth to display data on a map with plotly px.choropleth() and parameters like locations='ISO' and color_continuous_scale='matter'
Use plotly to display a whole bunch of lines at once (without doing a for loop like with matplotlib) with .line() and the color= parameter
Create Sunburst charts with plotly, e.g.
fig12 = px.sunburst(prize_orgs, path=['organization_country', 'organization_city', 'organization_name'],
values='nb_of_nobel_prizes', height=1200,
title='Nobel Prizes distributed by Research Institutions')
fig12.show()
- Create a histogram with Seaborn and the
.histplot() method, as well as giving a pre-defined style and override some of its values with rc={}, and changing the number of bars/bins with bins=, and more:
plt.figure(figsize=(8,4), dpi=200)
# using ticks style (so xticks appear) and overriding background color
with sns.axes_style('ticks', rc={'axes.facecolor': '#EAEAF2'}):
hist = sns.histplot(x=prize_per_age.winning_age, bins=30)
hist.set(xlabel='Age of Nobel Prize Laureates')
hist.grid(color='white', linestyle='--', axis='y')
# plt.xticks(ticks=hist_x_ticks)
plt.show()
Create a regression plot with Seaborn with a moving average line using the lowess=True parameter (statsmodels package must be installed though).
Create box plots with Seaborn and sns.boxplot()
Remove legend in Seaborn or matplotlib plots with plt.legend([],[], frameon=False
Use Seaborn's .lmplot() and show scatter and regression/best-fit lines across multiple categories using the hue, height, aspect and lowess parameters, either in one window or in multiple with col, col_wrap and row.
